We present the Recurrent Interface Network (RIN), a neural net architecture that allocates computation adaptively to the input according to the distribution of information, allowing it to scale to iterative generation of high-dimensional data. Hidden units of RINs are partitioned into the interface, which is locally connected to inputs, and latents, which are decoupled from inputs and can exchange information globally. The RIN block selectively reads from the interface into latents for high-capacity processing, with incremental updates written back to the interface. Stacking multiple blocks enables effective routing across local and global levels. While routing adds overhead, the cost can be amortized in recurrent computation settings where inputs change gradually while more global context persists, such as iterative generation using diffusion models. To this end, we propose a latent self-conditioning technique that "warm-starts" the latents at each iteration of the generation process. When applied to diffusion models operating directly on pixels, RINs yield state-of-the-art image and video generation without cascades or guidance, while being domain-agnostic and up to 10$\times$ more efficient compared to specialized 2D and 3D U-Nets.

Text-guided image editing can have a transformative impact in supporting creative applications. A key challenge is to generate edits that are faithful to input text prompts, while consistent with input images. We present Imagen Editor, a cascaded diffusion model built, by fine-tuning Imagen on text-guided image inpainting. Imagen Editor's edits are faithful to the text prompts, which is accomplished by using object detectors to propose inpainting masks during training. In addition, Imagen Editor captures fine details in the input image by conditioning the cascaded pipeline on the original high resolution image. To improve qualitative and quantitative evaluation, we introduce EditBench, a systematic benchmark for text-guided image inpainting. EditBench evaluates inpainting edits on natural and generated images exploring objects, attributes, and scenes. Through extensive human evaluation on EditBench, we find that object-masking during training leads to across-the-board improvements in text-image alignment -- such that Imagen Editor is preferred over DALL-E 2 and Stable Diffusion -- and, as a cohort, these models are better at object-rendering than text-rendering, and handle material/color/size attributes better than count/shape attributes.

Panoptic segmentation assigns semantic and instance ID labels to every pixel of an image. As permutations of instance IDs are also valid solutions, the task requires learning of high-dimensional one-to-many mapping. As a result, state-of-the-art approaches use customized architectures and task-specific loss functions. We formulate panoptic segmentation as a discrete data generation problem, without relying on inductive bias of the task. A diffusion model based on analog bits is used to model panoptic masks, with a simple, generic architecture and loss function. By simply adding past predictions as a conditioning signal, our method is capable of modeling video (in a streaming setting) and thereby learns to track object instances automatically. With extensive experiments, we demonstrate that our generalist approach can perform competitively to state-of-the-art specialist methods in similar settings.

尽管语言任务自然而然地以单个，统一的建模框架（即生成代币序列）表示，但在计算机视觉中并非如此。结果，对于不同的视力任务，不同的架构和损失功能的扩散。在这项工作中，我们表明，如果根据共享像素到序列界面进行配制，也可以统一一组“核心”计算机视觉任务。我们专注于四个任务，即对象检测，实例分割，关键点检测和图像字幕，所有这些任务都具有各种类型的输出，例如边界框或密集的掩码。尽管如此，通过将每个任务的输出作为具有统一界面的离散令牌的顺序，我们表明可以在所有这些任务上训练具有单个模型体系结构和损失功能的神经网络，而没有特定于任务的自定义。为了解决特定的任务，我们使用一个简短的提示作为任务说明，序列输出适应提示，以便它可以产生特定于任务的输出。我们表明，与成熟的特定任务模型相比，这种模型可以实现竞争性能。

医疗人工智能（AI）的最新进展已提供了可以达到临床专家水平绩效的系统。但是，当在与训练环境不同的临床环境中评估时，这种系统往往会证明次优的“分布式”性能。一种常见的缓解策略是使用特定地点数据为每个临床环境开发单独的系统[1]。但是，这很快变得不切实际，因为医疗数据很耗时，可以注释且昂贵[2]。因此，“数据有效概括”的问题给医学AI开发带来了持续的困难。尽管代表性学习的进展显示出希望，但并未对其好处进行严格的研究，特别是用于分布的设置。为了应对这些挑战，我们提出了RESEDIS，这是一种统一的代表学习策略，以提高医学成像AI的鲁棒性和数据效率。雷雷迪斯使用大规模监督转移学习与自我监督学习的通用组合，几乎不需要特定于任务的自定义。我们研究各种医学成像任务，并使用回顾性数据模拟三个现实的应用程序场景。 RESEDIS表现出明显改善的分布性能，而在强有力的基线上，诊断准确性相对相对提高了11.5％。更重要的是，我们的策略会导致对医学成像AI的强大数据有效的概括，并使用跨任务的1％至33％的重新培训数据匹配强有力的监督基线。这些结果表明，Repedis可以显着加速医学成像AI开发的生命周期，从而为医学成像AI提供了重要的一步，以产生广泛的影响。

生成时间连贯的高保真视频是生成建模研究中的重要里程碑。我们通过提出一个视频生成的扩散模型来取得这一里程碑的进步，该模型显示出非常有希望的初始结果。我们的模型是标准图像扩散体系结构的自然扩展，它可以从图像和视频数据中共同训练，我们发现这可以减少Minibatch梯度的方差并加快优化。为了生成长而更高的分辨率视频，我们引入了一种新的条件抽样技术，用于空间和时间视频扩展，该技术的性能比以前提出的方法更好。我们介绍了大型文本条件的视频生成任务，以及最新的结果，以实现视频预测和无条件视频生成的确定基准。可从https://video-diffusion.github.io/获得补充材料

培训细节和数据集对于筏等最新的光流模型有多重要？它们会概括吗？为了探索这些问题，而不是开发新的模型，我们将重新访问三个突出的模型，即PWC-NET，IRR-PWC和RAFT，并采用一组常见的现代培训技术和数据集，并观察到显着的性能增长，证明了重要性和普遍性这些培训细节。我们新训练的PWC-NET和IRR-PWC模型显示出惊人的改进，与Sintel和Kitti 2015 Benchmarks相比，最高30％的结果与原始发布的结果相比。他们的表现胜过2015年Kitti的最新流程1D，而推断过程中的速度快3倍。我们新训练的筏子在2015年的Kitti上获得了4.31％的成绩，比写作时所有已发表的光流方法更准确。我们的结果表明，分析光流方法的性能提高时，分离模型，训练技术和数据集的贡献的好处。我们的源代码将公开可用。

我们使用条件扩散模型介绍调色板，这是一种简单而一般的框架，可用于图像到图像到图像转换。在四个具有挑战性的图像到图像转换任务（着色，染色，un折叠和JPEG减压），调色板优于强大的GaN和回归基线，并建立了新的最新状态。这是在没有特定于任务特定的超参数调整，架构定制或任何辅助损耗的情况下实现的，展示了理想的一般性和灵活性。我们揭示了使用$ l_2 $与vs. $ l_1 $损失在样本多样性上的越来越多的影响，并通过经验架构研究表明自我关注的重要性。重要的是，我们倡导基于想象项目的统一评估协议，并报告包括预先训练的Reset-50的FID，成立得分，分类准确度的多个样本质量评分，以及针对各种基线的参考图像的感知距离。我们预计这一标准化评估协议在推进图像到图像翻译研究方面发挥着关键作用。最后，我们表明，在3个任务（着色，染色，JPEG减压）上培训的单个通用调色板模型也表现或优于特定于任务专家的专家对应物。

我们表明，级联扩散模型能够在类条件的想象生成基准上生成高保真图像，而无需辅助图像分类器的任何帮助来提高样品质量。级联的扩散模型包括多个扩散模型的流水线，其产生越来越多的分辨率，以最低分辨率的标准扩散模型开始，然后是一个或多个超分辨率扩散模型，其连续上追随图像并添加更高的分辨率细节。我们发现级联管道的样本质量至关重要的是调节增强，我们提出的数据增强较低分辨率调节输入到超级分辨率模型的方法。我们的实验表明，调节增强防止在级联模型中采样过程中的复合误差，帮助我们在256×256分辨率下，在128x128和4.88，优于63.02的分类精度分数，培训级联管道。 ％（TOP-1）和84.06％（TOP-5）在256x256，优于VQ-VAE-2。

In this paper, we propose a novel technique, namely INVALIDATOR, to automatically assess the correctness of APR-generated patches via semantic and syntactic reasoning. INVALIDATOR reasons about program semantic via program invariants while it also captures program syntax via language semantic learned from large code corpus using the pre-trained language model. Given a buggy program and the developer-patched program, INVALIDATOR infers likely invariants on both programs. Then, INVALIDATOR determines that a APR-generated patch overfits if: (1) it violates correct specifications or (2) maintains errors behaviors of the original buggy program. In case our approach fails to determine an overfitting patch based on invariants, INVALIDATOR utilizes a trained model from labeled patches to assess patch correctness based on program syntax. The benefit of INVALIDATOR is three-fold. First, INVALIDATOR is able to leverage both semantic and syntactic reasoning to enhance its discriminant capability. Second, INVALIDATOR does not require new test cases to be generated but instead only relies on the current test suite and uses invariant inference to generalize the behaviors of a program. Third, INVALIDATOR is fully automated. We have conducted our experiments on a dataset of 885 patches generated on real-world programs in Defects4J. Experiment results show that INVALIDATOR correctly classified 79% overfitting patches, accounting for 23% more overfitting patches being detected by the best baseline. INVALIDATOR also substantially outperforms the best baselines by 14% and 19% in terms of Accuracy and F-Measure, respectively.